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ABSTRACT 

As part of the U.S. Department of Commerce, the National Institute 
of Standards and Technology (NIST) is examining new approaches 
to increase the competitiveness of smaller U.S.-based manufacturers. 
Collaborative robotics is one way to increase automation, leverag¬ 
ing the specialized skills of human workers along with the increased 
repeatability and throughput of robots. However, in order for hu¬ 
mans to efficiently work alongside robots, new types of interfaces 
are needed to streamline interactions. Augmented reality, especially 
wearable devices, are a new approach that can enable task training, 
hands-free interactions, and increased safety through situational 
awareness. This paper will explore some preliminary approaches 
to integrating augmented reality for a collaborative manufacturing 
environment. 
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1 INTRODUCTION 

Automation, particularly the use of robotics, is becoming more 
prevalent among the larger U.S. manufacturers [1, 10], and this trend 
has been predicted to continue growing exponentially in the near 
future [12]. Recent trends show robots are increasingly being used 
in small- and medium-sized enterprises (SMEs) as well, principally 
due to decreasing costs and advances in safety [8], In particular, 
collaborative robots offer SMEs the advantage of decreased training 
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overhead and increased safety without requiring changes in the 
work environment to accommodate cages, thus leading to greater 
flexibility and multi-purpose use [7]. 

Current off-the-shelf collaborative robots, however, are gener¬ 
ally only collaborative in the sense that they are expected to be 
safety-rated to operate near humans, incorporating force-limited 
motions or collision avoidance. To enable truly collaborative tasks 
that can leverage the abilities of existing human workers, not only 
do robots need to be aware of their environment and human part¬ 
ners), but human-facing interfaces need to evolve as well to enable 
real-time, two-way communication. Traditional teach pendant in¬ 
terfaces do not typically allow for easy or convenient use while 
also performing a task. Likewise, teach pendants make it difficult to 
obtain feedback or task status from the robot in an easy and efficient 
manner. New types of interfaces need to be developed to simplify 
interactions and incorporate system diagnostics, prognostics, and 
situation awareness (SA) before truly collaborative robotics can 
become a reality. 

Augmented Reality (AR) technology has advanced rapidly in 
recent years. Now that such devices are widely available commer¬ 
cially, their acceptance is becoming increasingly common as their 
reliability improves. This increased acceptance and reliability en¬ 
ables integrators to leverage their use as human-robot interaction 
(HRI) interfaces for industrial applications. In particular, wearable 
AR devices have been shown to increase situational awareness (SA) 
and lead to better task performance [3], as well as grounding in the 
robot state [4]. 

Most AR devices can be split into two categories: “see-through” 
or monitor-based displays [9]. As the name implies, see-through dis¬ 
plays project information onto a transparent medium that permits 
users to retrieve information without having to divert attention 
from the principal area of focus. Monitor-based displays, in con¬ 
trast, require the user to focus attention on a separate display or 
interface through which the world is observed. This paper will 
explore multiple types of AR interfaces and their differing uses as 
industrial robot interfaces. As prototype interfaces are developed, 
future work will allow the collection of user data and enable further 
conclusions about the utility of AR devices for HRI. 

2 WEARABLE DEVICES 

In the category of see-through, wearable AR technology, there are 
generally two types of devices available: headsets and smart glasses. 
Smart glasses, such as Google Glass or Vuzix Blade, generally have 
one small screen projected into the user’s direct or peripheral view. 
Notably, the smaller form factor of these devices makes it difficult 
to incorporate advanced sensing beyond a single camera and mi¬ 
crophone. Natural language control is possible, but may not be as 
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useful in a loud manufacturing environment. This limits the user’s 
interaction with the real world through virtual objects (“mixed 
reality”) as well as the user’s communication with the robot, and 
thus limits the application of these devices in HRI. 

For the purpose of industrial collaborative robotics, AR headsets 
such as Microsoft’s HoloLens or the Meta 2 from Meta Company are 
more frequently being considered. There are also manufacturing- 
specific devices being developed, such as the DAQRI Smart Helmet, 
that are directly integrated into personal protective equipment 
(PPE) so as to ensure safety even while using immersive interfaces. 
Untethered headsets with wireless communication capabilities are 
ideal for performing manufacturing tasks while communicating 
with a robot partner. Additionally, these devices contain multiple 
cameras and inertial measurement units (IMUs) to perform head 
tracking and hand gesture recognition. The ability to project larger, 
three-dimensional holograms, as well as perform object recognition 
and other perception techniques, makes these AR devices promising 
for HRI applications. 

2.1 Task Augmentation 

Smaller manufacturers are often retasking their industrial robots 
for new products or applications [5, 11], When retasking, human 
workers must be trained and industrial robots must be programmed 
before new production can begin. AR headsets can be useful on 
both sides of this equation by performing task-specific environment 
augmentations. 

While wearing an AR headset, task training for the operator 
could be performed in an abbreviated training session or poten¬ 
tially even on-the-job. For example, relevant parts for an assembly 
can be identified by the headset and highlighted in the kit tray 
using a holographic overlay. Instructions can appear to the user 
textually, graphically, or verbally, depending on the situation or 
the user’s preferences. The robot’s part of the collaborative task 
can be identified by the headset as well, allowing the human to 
train it using the AR control interface, or other options such as 
kinesthetic learning or teaching by demonstration. In this way, both 
task partners can benefit from the AR interface. 

Task augmentation can also be performed as a method of error 
detection. In the case that the wrong part is chosen for an assembly, 
for instance, the headset’s perception system could detect a mis¬ 
match and alert the operator. This should save time down the line 
in the quality assurance process. 

2.2 Robot Feedback 

Similar to error detection, robot fault detection can be integrated 
into an AR interface. The robot constantly reports its status to the 
headset, which can monitor several relevant parameters such as 
joint torque limits or Cartesian speed. Should a given parameter 
begin to approach its limit, the headset can alert the operator to 
the issue and display potential causes or methods of handling the 
issue. 

The robot can also use communication with the headset to query 
its human teammate. There may be issues where the robot’s per¬ 
ception system is occluded or unsure which object is the goal, or 
an object may be out of the robot’s reach. The robot can transmit 
these types of questions to the user - in text, visually, or verbally 


through natural language processing - to get assistance and make 
use of human knowledge or flexibility. 

2.3 Situation Awareness 

In addition to robot fault reporting as discussed above, there are 
multiple ways AR headsets and interfaces can improve operator 
situation awareness and grounding. One such option is for the robot 
to inform the operator of its next action, either through a virtual 
model (hologram) or by highlighting objects in the environment. 
This would allow the human teammate to adjust their actions ac¬ 
cordingly, helping accomplish preconditions for the robot’s next 
task, and potentially improving safety as the human knows the 
robot’s intent. The operator can also intervene if the robot intends 
to perform the wrong task or has the potential of colliding with 
other objects. In this way, the collaborative task can take advantage 
of the robot’s efficiency while still incorporating human oversight. 
Additionally, the human knows more about the robot’s current 
state, and potentially feels more comfortable working directly with 
the robot. 

3 PORTABLE DEVICES 

On the opposite side of the spectrum from see-through or wearable 
devices are simpler devices with display screens and rear cameras 
(e.g., tablets and smart phones). These consumer devices are widely 
available for a relatively low price point, and can execute AR soft¬ 
ware comparable to headsets without the low screen real estate of 
smart glasses. External camera feeds can also be used with any com¬ 
puter, but are not significantly different from existing interfaces, 
so they will not be discussed here. Tablets are particularly inter¬ 
esting due to the potential of touchscreen interfaces. The recent 
prevalence of touchscreen devices means that interfaces on similar 
devices will require less training for novice users. The portable 
nature of these devices also gives them further utility for use in 
technical support and diagnostics applications. 

3.1 Diagnostics & Prognostics 

An example scenario we have envisioned for walk-up diagnostics 
is as follows: A technician is making the rounds of the industrial 
robots on a factory floor. At each robot, they use their tablet camera 
to scan an AR target (an image or other symbol, defined ahead 
of time). The scan connects them to the particular robot they are 
examining, via the factory’s local wireless network, and launches 
the AR application. The technician can now point the camera at the 
robot, which is overlaid with a virtual model displaying the current 
robot status - for example, the torque at every joint. Problem areas 
reported by the robot will be highlighted in the AR view. Addition¬ 
ally, the technician can easily control the robot via the touchscreen 
interface to test operations or access areas in need of service. 

Figure 1 shows a prototype tablet interface that incorporates 
several of the features mentioned above. A virtual model of the 
robot appears after scanning the image target shown in the photo 
(here a printed photo of the robot in question). Joint-specific con¬ 
trols show on the right side of the screen, along with options for 
communicating with the non-virtual robot. The robot’s current 
configuration is updated live through both the virtual avatar and 
the joint sliders. Through this type of interface, a technician can 
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Figure 1: Prototype tablet interface using Augmented Real¬ 
ity technology. 

observe and monitor the robot, as well as control it and receive 
relevant status information of the robot’s current task. 

4 EVALUATION & FUTURE WORK 

Before new technology will be accepted by manufacturers, it must 
be thoroughly tested for performance, reliability, and safety. SMEs, 
in particular, have lower risk-tolerance, which may prevent them 
from investing in technologies that have not been exhaustively 
evaluated, or vetted through broad industry use [13]. Therefore, we 
intend to conduct a comprehensive study into the benefits of AR 
interfaces as compared to current methods. 

Currently, several versions of AR interfaces for HRI are in devel¬ 
opment. These include headset, tablet, and touchscreen desktop ap¬ 
plications. Once development is completed, user studies are planned 
to compare these interfaces to more traditional interfaces such as a 
teach pendant. This will allow us to collect user data in the form 
of both objective and subjective measures. For the former, these 
might include performance metrics such as task completion time 
or measures of success. For the latter, participants will be asked to 
evaluate each interface according to their preferences, answering 
post-task survey questions on a Likert scale. In addition, measures of 
mental workload will be collected from the Task-Load Index (TLX) 
developed by the National Aeronautics and Space Administration 
(NASA) [6], and task situation awareness will be measured from 
the situation awareness rating technique (SART) or the situation 
awareness global assessment technique (SAGAT) tests [2]. 

Collecting a wide variety of metrics and surveying a large pool of 
participants will allow us to draw conclusions about the strengths 
and weaknesses of each type of interface. Ultimately, our goal is also 
to evaluate which metrics are most important for HRI interfaces. 
In this way, we can help guide the creation of commercial HRI 
interfaces that have proven performance, are thoroughly safety 
tested, and held to accepted standards - thus more likely to be 
adopted in industrial settings. 


5 CONCLUSIONS 

As manufacturers move towards greater automation, new types 
of technology are needed to create a truly collaborative working 
environment. Augmented Reality can be a very useful tool for 
making human workers more effective, particularly for human- 
robot collaboration. As robot interfaces, AR devices can provide 
not only effective robot control, but also task training, feedback, 
and better SA. The process of robot diagnostics, root cause analysis, 
health monitoring, and prognostics can also be improved through 
the use of AR. In the future, we expect to quantitatively evaluate 
the potential performance benefits of AR used for HRI. 

DISCLAIMER 

Certain commercial equipment, instruments, or materials are iden¬ 
tified in this paper to foster understanding. Such identification 
does not imply recommendation or endorsement by the National 
Institute of Standards and Technology, nor does it imply that the 
materials or equipment identified are necessarily the best available 
for the purpose. 
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